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REMARKS 

This Preliminary Amendment cancels original claims 1 to 14 in the underlying 
PCT Application No. PCT/DE03/03744 and adds new claims 15 to 34. The new claims 
conform the claims to the U.S. Patent and Trademark Office rules and do not add new matter 
to the application. 

In accordance with 37 C.F.R. § 1.125, the Substitute Specification (including 
the Abstract, but without the claims) contains no new matter. The amendments reflected in 
the Substitute Specification (including the Title and Abstract) are to conform the 
Specification and Abstract to U.S. Patent and Trademark Office rules or to correct 
informalities. As required by 37 C.F.R. § 1.121 and § 1.125, a Marked Up Version Of The 
Substitute Specification comparing the Specification of record and the Substitute 
Specification also accompanies this Preliminary Amendment. In the Marked Up Version, 
underlining indicates added text and "strike-throughs" and double-brackets indicate deleted 
text. Approval and entry of the Substitute Specification (including Abstract) are respectfully 
requested. 

Also enclosed is a translated copy of the International Search Report dated 
March 5, 2004. The Search Report includes a list of documents that were considered by the 
Examiner in the underlying PCT application. 



It is asserted that the subject matter of the present application is new, non- 
obvious, and useful. Prompt consideration and allowance of the application are respectfully 
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METHOD FOR ADJUSTING AN IMAGE SENSOR 
Background Information F IELD OF THE INV ENTION 

The present invention relates to a method and a processing unit for adjusting the 
characteristic curve of the exposure sensitivity of an image sensor, the image sensor being 
located in particular in a motor vehicle. 

5 BACKGROUND INFORMATION 

Outdoor scenes often have a broad range of different radiation intensities. In particular in 
traffic scenes in the motor vehicle field, frequently a great difference in radiation intensities 
occurs. This becomes clear for example when a motor vehicle approaches a tunnel entrance in 
daylight. While the surroundings of the motor vehicle outside the tunnel are bright in the 

10 sunlight, often only weak lighting is found in the tunnel itself. With the plan to equip motor 
vehicles with image sensors which register the surroundings of the motor vehicle, it is 
desirable for the image sensors to process this wide range of different radiation intensities. In 
the conventional regulation of an image sensor, the image sensor is adapted to the lighting 
conditions through appropriate choice of the parameters of gain, offset, integration time, 

15 and/or aperture. In newer developments of image sensors, the exposure sensitivity does not 
follow a rigidly set linear or logarithmic characteristic curve, but rather the characteristic 
curve may be adjusted individually in particular in single linear segments. For example, a 
CMOS image sensor having an adjustable characteristic curve with linear segments is known 
from US 6,318.681 of February 19. 2002. discussed in U.S. Patent No. 6.348.681. Reference 

20 to a method and a processing unit for attaining high information content of generated images 
is lacking in US U.S. Patent No. 6,348,681 . 

Advantag e s of the Invention 
SUMMARY OF THE I NVENTION 
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The method described below for adjusting the characteristic curve of the exposure sensitivity 
of at least one pixel of at least one image sensor has the advantage that the generated image 
information has high information content and a broad dynamic range. The method described 
below is particularly advantageous in outdoor use of the at least one image sensor, since 
5 outdoors a plurality of different radiation intensities may occur simultaneously in a scene. In 
particular, the method described below is suitable for use in a motor vehicle, since with this 
method, for example, sections of streets which are in full sunlight and simultaneously have 
areas of shadow may be mapped with high information content of the image information. In 
an advantageous manner, the method described below utilizes the additional degrees of 
10 freedom of a characteristic curve of the exposure sensitivity having segments of functions, in 
particular continuous and/or linear functions, so that the at least one image sensor captures a 
maximum of information in its images about the scene being mapped at that time. 

It is advantageous to approximate the characteristic curve of the exposure sensitivity to the 
optimal characteristic curve of the exposure sensitivity through at least one numerical 
15 approximation method and/or at least one segmenting method. In particular, approximation 
through at least one segmenting method has the advantage that the at least one segmenting 
method contributes to a simple implementation of the method on a microprocessor. 

Particularly advantageous is adjustment of the characteristic curve of the exposure sensitivity, 
the characteristic curve being formed in segments of linear functions. Characteristic curves of 
20 the exposure sensitivity having segments of linear functions contribute to the characteristic 
curves being easily realizable in the image sensor, and also enable a simple algorithmic 
implementation of the method described below in the processing unit. 

It is advantageous to adjust the characteristic curve of the exposure sensitivity of at least one 
pixel of at least one image sensor as a function of image signals from at least two image 
25 sensors, in particular from at least one stereo camera which is made up of at least two 

sensors. This procedure has the advantage that by averaging the image signals the precision 
and robustness of the method for adjusting the exposure sensitivity of at least one pixel of at 
least one image sensor is increased. 

The described advantages apply correspondingly to the method described below and/or the 
30 processing unit described below and/or the computer program described below. 

A computer program having ajprogram code meafts arrangement for carrying out all the steps, 
or at least the essential steps of the described method, when the program is executed on a 
computer, is advantageous. The use of a computer program facilitates quick and cost- 
effective adaptation of the method, for example to different image sensors. 
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Additional advantages derive from the following description of exemplary embodiments with 
reference to the figures , and from th e d e p e nd e nt claims . 

Drawing 

Th e pr e s e nt inv e ntion is e xplain e d in gr e at e r d e tail b e low on th e basis of th e e mbodiments 
5 illustrat e d in the drawing. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 shows a diagram with characteristic curves of the exposure sensitivity of an 

image sensor^ 

10 Figure 2 shows a diagram with characteristic curves of the exposure sensitivity of an 

image sensor as a function of the gain and the offset^ 

Figure 3 shows a block diagram of the pr e f e rr e d ^ exemplary embodiment. 

15 Detailed D e scription of Exemplary Embodiments 
DETAILED DESCRIPTION 

A method and a processing unit for adjusting the characteristic curve of the exposure 
sensitivity of at least one pixel of at least one image sensor in a motor vehicle are described 
below. In th e pref e rr e d an exemplary embodiment, the optimal characteristic curve of the 

20 exposure sensitivity is determined from the histogram of the gray values of at least one image 
as a function of image signals from at least one image sensor. The freely definable 
characteristic curve of the image sensor having linear segments is chosen so that it agrees at 
least approximately with the optimal characteristic curve. Along with the size and/or the 
position of the characteristic curve of the exposure sensitivity, alternatively or in addition the 

25 shape of the characteristic curve is adjusted. 

Figure 1 shows a diagram with three characteristic curves 14, 16, 18 of the exposure 
sensitivity of a pixel of an image sensor. Exposure 10 is plotted on the abscissa. Exposure 10 
is plotted here in freely defined, virtual units of 0 to 100, exposure 10 being for example a 
measure of the radiation intensity or the illumination intensity. Here the radiation intensity is 
30 a physical value which indicates the radiation power per unit area, while the illumination 
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intensity is the corresponding photometric value using the unit lux, which takes into account 
the spectral sensitivity of the human eye. Output signal 12 of the pixel of the image sensor is 
plotted on the ordinate as a freely defined, virtual unit from 0 to 1000. Output signal 12 may 
be present as a digital signal or an analog signal. As a first form of the characteristic curve of 
5 the exposure sensitivity, a linear characteristic curve 14 of the exposure sensitivity is plotted 
in Figure 1. The diagram in Figure 1 also shows a logarithmic characteristic curve 16 of the 
exposure sensitivity and a characteristic curve 18 of the exposure sensitivity in linear 
segments. 

Figure 2 shows a diagram with four characteristic curves 20, 22, 24, 26 of the exposure 

10 sensitivity in linear segments of a pixel of an image sensor as a function of the gain and/or the 
offset. Corresponding to Figure 1, exposure 10 is plotted on the abscissa in freely defined, 
virtual units from 0 to 100. Output signal 12 of the pixel of the image sensor is plotted on the 
ordinate as a freely defined, virtual unit from 0 to 2000. In first characteristic curve 20 of the 
exposure sensitivity, the gain is 1 and the offset 0. In contrast, in second characteristic curve 

15 22 of the exposure sensitivity the offset has a value of +100, while in third characteristic 

curve 24 of the exposure sensitivity it has a value of -100. In both second characteristic curve 
22 of the exposure sensitivity and third characteristic curve 24 of the exposure sensitivity, the 
gain is 1. In fourth characteristic curve 26 of the exposure sensitivity, the gain is 2 with an 
offset of 0. A change in the offset shifts the characteristic curve of the exposure sensitivity in 

20 the ordinate direction, while a change in the gain and/or a change in the integration time 
expands or compresses the characteristic curve of the exposure sensitivity in the ordinate 
direction. A freely configurable characteristic curve of the exposure sensitivity in linear 
segments, such as for example first characteristic curve 20 of the exposure sensitivity in 
Figure 2, results in more degrees of freedom when adjusting at least one pixel of the image 

25 sensor, since here single segments can be adjusted individually. Along with the size and 
position of characteristic curve 20, 22, 24, 26 of the exposure sensitivity, this makes it 
alternatively or additionally possible to change the shape of characteristic curve 20, 22, 24, 
26 of the exposure sensitivity. 

Figure 3 shows a block diagram of the preferr e d exemplary embodiment, made up of an 
30 image sensor 40 and a processing unit 42 having various modules 50, 52, 54, 56. In the 

pref e rr e d ^ exemplary embodiment, image sensor 40 is located behind the windshield of a 
motor vehicle in the area of the interior rearview mirror. In a variant of the pr e f e rr e d 
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exemplary embodiment, image sensor 40 is built into the bumper and/or a side mirror and/or 
the taillight. Image sensor 40 is aligned in such a way that the image capture zone includes at 
least part of the surroundings of the motor vehicle. In the preferred exemplary embodiment, a 
black/white CMOS image sensor having a resolution of 640x480 pixels with 8-bit gray scale 
5 resolution is used as image sensor 40. Image sensor 40 has a freely configurable 

characteristic curve of the exposure sensitivity in linear segments, the number of segments 
and/or the position of the segments and/or the size of the individual segments and/or at least 
one parameter of the linear functions in a segment being adjustable. Alternatively or in 
addition, the gain and/or the offset and/or the integration time are adjusted. In the pr e f e rr e d 

10 exemplary embodiment, the setting of the characteristic curve of the exposure sensitivity is 
adjusted jointly for all pixels of image sensor 40. Individual modules 50, 52, 54, 56 of 
processing unit 42 are implemented in the pr e f e rred exemplary embodiment in a digital 
microprocessor, which implements the functions of modules 50, 52, 54, 56 described below 
as programs, subprograms, or program steps. In one variant there are at least two 

15 microprocessors, to which individual modules 50, 52, 54, 56 are distributed. Of course it is 
also possible to distribute the individual modules to discrete processing units with their own 
computing capabilities. 

The image signals generated by image sensor 40 are transmitted over signal line 44 to 
processing unit 42. The image signals are transmitted electrically and/or optically and/or by 

20 radio. From the incoming image signals, i.e., the camera images, the histogram of the gray 

values is determined in histogram determination module 50. The histogram of the gray values 
indicates the frequency of the individual gray values of an image or image detail. In the 
pr e f e rr e d exemplary embodiment, the histogram of the gray values is determined from an 
overall image. For each gray value, the associated illumination intensity is determined in 

25 freely definable, virtual units in histogram determination module 50 as a function of the 
parameters set, such as the gain and/or the offset and/or the integration time, and/or the 
parameters for adjusting the characteristic curve of the exposure sensitivity, such as the 
number of segments of the characteristic curve of the exposure sensitivity, and/or the position 
of the segments of the characteristic curve of the exposure sensitivity, and/or the size of the 

30 segments of the characteristic curve of the exposure sensitivity, and/or the at least one 
parameter of the linear functions. 
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In module 52.for determining the optimal characteristic curve, the optimal characteristic 
curve is calculated as a function of the histogram of the gray values determined in preceding 
histogram determination module 50. With N gray values in the histogram, having respective 
frequencies hj and associated illumination intensities Oj, the result is a table of N elements, 
5 whose entries i are calculated for example from 



where Gmax designates the maximum depictable gray value. The calculated function for 
mapping illumination intensities <D on gray values g represents the optimal characteristic 
curve of the exposure sensitivity according to information theory for the given histogram, and 
10 is an optimal characteristic curve of the exposure sensitivity. 

The optimal characteristic curve of the exposure sensitivity is distinguished basically by the 
fact that after adjustment of the characteristic curve of the exposure sensitivity of the image 
sensor, which is approximated to the determined optimal characteristic curve of the exposure 
sensitivity, the frequency of the individual gray values in the histogram of the image signals, 

15 i.e., of the camera image of the image sensor, is approximately constant, and/or the gray value 
density of the histogram of the image signals, i.e., of the camera image of the image sensor, is 
approximately constant. The frequency of a gray value here designates the number of pixels 
within the camera image that have this gray value based on the total number of pixels. A 
constant frequency of the gray values within a histogram of an image is referred to as a 

20 uniformly distributed histogram. The gray value density designates the sum of frequencies 
h(gi) of gray values gi in an interval Ag of gray values in reference to this interval Ag: (H 



In module 54 for approximating the characteristic curve of the exposure sensitivity, the 
characteristic curve of the exposure sensitivity in linear segments of the pixels of the image 

25 sensor is now chosen so that it agrees as precisely as possible with the optimal characteristic 
curve obtained. In the preferred exemplary embodiment, this is performed in module 54 for 
approximating the characteristic curve of the exposure sensitivity, through a numerical 
minimization of the quadratic separation between the optimal characteristic curve of the 
exposure sensitivity and the characteristic curve in linear segments, while varying the 

30 parameters. 




h(gi))/Ag. 
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Alternatively or in addition, in a variant of the pr e f e rr e d exemplary embodiment, the 
segmenting method described in Hwa-Cho Yi: Sensordatenauswertung mit Fuzzy-Logik 
fefuer das automatisierte Entgraten [Sensor data evaluation with fuzzy logic for automated 
deburring] (Munich: Hanser-Verlag, 1993, TU-Berlin dissertation, pp. 76-79) is used. This 
5 segmenting method operates according to the iterative final point method or the principle of 
the search for the longest segment. The determined parameters, such as the gain and/or the 
offset and/or the integration time and/or number of segments of the characteristic curve of the 
exposure sensitivity and/or the position of the segments of the characteristic curve of the 
exposure sensitivity and/or the size of the segments of the characteristic curve of the exposure 
10 sensitivity, and/or at least one parameter of the linear functions, are transmitted from module 
54 for approximating the characteristic curve of the exposure sensitivity to module 56 for 
generating the adjustment signals. 

In module 56 for generating the adjustment signals, the at least one corresponding adjustment 
signal is determined from the determined parameters, and the at least one adjustment signal is 

15 transmitted over signal line 46 electrically and/or optically and/or by radio to image sensor 
40. In image sensor 40, the characteristic curve of the exposure sensitivity is adjusted as a 
function of at least one adjustment signal. In the pr e ferr e d exemplary embodiment, the 
characteristic curve of the exposure sensitivity takes effect in the image that follows 
determination of the optimal characteristic curve of the exposure sensitivity. Alternatively or 

20 in addition, in additional variants the characteristic curve of the exposure sensitivity takes 
effect in the same image and/or in the next image but one and/or in at least one additional 
image. 

The use of the method according to the present invention and/or of the processing unit 
according to the present invention and/or of the computer program according to the present 

25 invention for adjusting the characteristic curve of the exposure sensitivity of at least one pixel 
of at least one image sensor becomes clear when the at least one image sensor receives an 
incoming image having a wedge of gray values, the wedge of gray values being made up of 
two segments having a different gradient of the gray values. In otherwise adjusted and/or 
controlled and/or regulated image sensors having a characteristic curve in linear segments, 

30 with variable overall illumination of the gray value wedge, an apparent contour caused by the 
bent characteristic curve of the exposure sensitivity appears, which does not agree with the 
segment boundary of the gray value wedge. In contrast, through the method according to the 
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present invention and/or the processing unit according to the present invention and/or the 
computer program according to the present invention, this apparent contour will lie exactly 
on or at least very close to the separation line, even if the overall illumination is changed. 

The described method and/or processing unit and/or computer program are not restricted to 
adjusting the characteristic curve of the exposure sensitivity of an image sensor in a motor 
vehicle. Rather, the described approach using the corresponding features is also usable 
outside of motor vehicle technology, moreover with more than one image sensor. 
Furthermore, the described method and/or processing unit and/or computer program is not 
restricted to adjusting the characteristic curve of the exposure sensitivity. Rather, the 
approach is also suitable in particular for controlling and/or regulating the characteristic curve 
of the exposure sensitivity. 

In a variant of the described method and/or the processing unit and/or the computer program, 
the characteristic curve of the exposure sensitivity of at least one pixel of at least one image 
sensor is adjusted as a function of at least one image detail from at least one image sensor. A 
15 first embodiment includes the provision that the image segment is specifiable manually by at 
least one user, while in an additional embodiment the image detail is determined 
automatically by the processing unit. 

Another variant contains the provision, alternatively or in addition, that the characteristic 
curve of the exposure sensitivity of at least one pixel of at least one image sensor is adjusted 
as a function of image signals, in particular of at least one image and/or at least one image 
detail, from at least two or more image sensors. Alternatively or in addition, the adjustment of 
the characteristic curve of the exposure sensitivity of at least one pixel of at least one image 
sensor is performed as a function of image signals from at least one stereo camera. The stereo 
camera is made up here of at least two image sensors, which record essentially the same 
scene. For example, a common histogram of the gray values is calculated here from the 
images and/or image details and is used to determine the optimal characteristic curve of the 
exposure sensitivity. 

In general, the described procedure is not restricted to a black/white CMOS image sensor 
having a resolution of 640x480 pixels with 8-bit gray scale resolution. Rather, the method 
30 and/or the processing unit and/or the computer program are suitable for all types of image 
sensors. The only prerequisite is that the image sensors have a configurable characteristic 
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curve of the exposure sensitivity, having segments with functions, in particular continuous 
and/or linear functions. For example, in one variant a CCD image sensor is used. In another 
variant, alternatively or in addition, an image sensor having a logarithmic characteristic curve 
in segments is used. In addition, the described approach using the corresponding features is 
5 also usable with at least one color image sensor. Alternatively or in addition, in another 

variant the characteristic curve of the exposure sensitivity of at least one pixel of at least one 
image sensor is adjusted. 



MARKED UP VERSION OF 
SUBSTITUTE SPECIFICATION 



9 



Abstoe* ABSTRACT OF THE DISCLOSURE 

A method and a processing unit for adjusting the characteristic curve of the exposure 
sensitivity of at least one pixel of at least one image sensor in a motor vehicle are described. 
The optimal characteristic curve of the exposure sensitivity is determined from the histogram 
5 of the gray values of at least one image as a function of image signals from at least one image 
sensor. The freely definable characteristic curve of the image sensor having linear segments is 
chosen so that it agrees at least approximately with the optimal characteristic curve. Along 
with the size and/or the position of the characteristic curve of the exposure sensitivity, 
alternatively or in addition the shape of the characteristic curve is adjusted. 

10 

(Figur e 3) 
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